Circumventing Negative Transfer via Cross Generative Initialisation

Negative transfer – a special type of transfer learning – refers to the interference of the previous knowledge with new learning. In this research, through an empirical study, we demonstrate the futile defence to the negative transfer via conventional neural network based transfer techniques, i.e., mid-level feature extraction and knowledge distillation. Under a finer specification of transfer learning, we speculate the real culprits of negative transfer are the incongruence on task and model complexity and the ordering of learning. Based on this speculation, we propose a tentative transfer learning technique, i.e., cross generative initialisation, to sidestep the negative transfer. The effectiveness of cross generative initialisation was evaluated empirically. 1 UNAVAILING KNOWLEDGE TRANSFER IN NEGATIVE TRANSFER In inductive transfer learning (Pan & Yang, 2010), learning a target task is benefited by the transferred knowledge from a previous learned task, i.e., a source task. In neural network based transfer learning, the extracted mid-level features are served as the knowledge to transfer. The production of these transferable features is accomplished by running forward propagation of a trained neural network (Oquab et al., 2014). To optimise the usage of extracted features in transfer learning, Hinton et al (Hinton et al., 2015) proposed a knowledge distillation technique to allow the compression of a cumbersome model to a compact one. The distilled knowledge, i.e., the cross entropy loss of a learned neural network, is augmented to fine-tune a to-be-learned neural network. However, both techniques are futile in a special case of transfer learning: negative transfer. Negative transfer – a term borrowed from cognitive science – occurs in a situation where the prior learning of a source task interferes with the later learning of a target task (Pan & Yang, 2010). To formalise our discussion on negative transfer, considering two sequentially to-be-learned tasks with varying degrees of complexity; e.g., T1 (a complex task) and T2 (a simple task), we then assign two corresponding models, e.g., M1 (a cumbersome model) and M2 (a compact model) to learn the foregoing tasks. Dependent upon the congruence on task and model complexity and the learning sequence, there are four distinctive transfer learning cases, i.e., T1D1 → T2M2; T1M2 → T2M1; T2M1 → T1M2; T2M2 → T1M1. 1 As demonstrated in Table 1, negative transfer can only be circumvented in which the congruence on model and task complexity is high, and the ordering of learning is followed as complex to simple, i.e., T1M1 → T2M2. In other three cases, both mid-level feature extraction and knowledge distillation techniques failed to sidestep the negative transfer. → denotes the direction of knowledge transfer, e.g., T1D1 → T2M2 means the knowledge is extracted from a prior learning of a complex task through a cumbersome model, then transferred to assist the learning of a simple task through a compact model.